Usability Testing of an Electronic Patient-Reported Outcome System for Survivors of Critical Illness

BACKGROUND: Web-based electronic patient-reported outcomes (ePRO) measures are increasingly used to facilitate patient-centered health assessments. However, it is unknown if ePRO completion is feasible for recently ill intensive care unit (ICU) survivors and their families. OBJECTIVE: To develop and evaluate the usability of a novel ePRO system (ePRO to Support People and Enhance Recovery [ePROSPER]) among ICU survivors and their families within an ongoing clinical trial. METHODS: Paper-based PROs were iteratively adapted to electronic forms (ePROs). Then, the usability of ePROSPER was assessed among 60 patients, their family members, and PRO and programming experts via questionnaires (eg, Systems Usability Scale), "think aloud" open-ended feedback, task completion times, and error rates. RESULTS: Input from patients and their families was used to incorporate user-experience modifications into ePROSPER. This feedback also led to inclusion of automated reminders for questionnaire completion and real-time alerts for staff triggered by high symptom levels. Median usability scores increased over testing cycles from 40 to 73 to 95, nearing the maximum score and showing excellent usability. All users completed ePROSPER within 20 minutes; 87% preferred it to a written version. ePROSPER was then implemented in a clinical trial without data errors. CONCLUSIONS: Automated ePRO systems can be successfully integrated in a post-ICU clinical trial setting. The value of integrating such systems in direct clinical care should be assessed in future studies

Practical dyspnea assessment: relationship between the 0–10 numerical rating scale and the four-level categorical verbal descriptor scale of dyspnea intensity

Context—Measurement of dyspnea is important for clinical care and research. Objectives—To characterize the relationship between the 0–10 Numerical Rating Scale (NRS) and four-level categorical Verbal Descriptor Scale (VDS) for dyspnea assessment. Methods—This was a substudy of a double-blind randomized controlled trial comparing palliative oxygen to room air for relief of refractory breathlessness in patients with life-limiting illness. Dyspnea was assessed with both a 0–10 NRS and a four-level categorical VDS over the one-week trial. NRS and VDS responses were analyzed in cross section and longitudinally. Relationships between NRS and VDS responses were portrayed using descriptive statistics and visual representations. Results—Two hundred twenty-six participants contributed responses. At baseline, mild and moderate levels of breathlessness were reported by 41.9% and 44.6% of participants, respectively. NRS scores demonstrated increasing mean and median levels for increasing VDS intensity, from a mean (SD) of 0.6 (±1.04) for VDS none category to 8.2 (1.4) for VDS severe category. The Spearman correlation coefficient was strong at 0.78 (P < 0.0001). Based on the distribution of NRS scores within VDS categories, we calculated test characteristics of two different cutpoint models. Both models yielded 75% correct translations from NRS to VDS; however, Model A was more sensitive for moderate or greater dyspnea, with fewer misses downcoded. Conclusion—There is strong correlation between VDS and NRS measures for dyspnea. Proposed practical cutpoints for the relationship between the dyspnea VDS and NRS are 0 for none, 1–4 for mild, 5–8 for moderate, and 9–10 for severe

Development and usability testing of a Web-based decision aid for families of patients receiving prolonged mechanical ventilation

BackgroundWeb-based decision aids are increasingly important in medical research and clinical care. However, few have been studied in an intensive care unit setting. The objectives of this study were to develop a Web-based decision aid for family members of patients receiving prolonged mechanical ventilation and to evaluate its usability and acceptability.MethodsUsing an iterative process involving 48 critical illness survivors, family surrogate decision makers, and intensivists, we developed a Web-based decision aid addressing goals of care preferences for surrogate decision makers of patients with prolonged mechanical ventilation that could be either administered by study staff or completed independently by family members (Development Phase). After piloting the decision aid among 13 surrogate decision makers and seven intensivists, we assessed the decision aid’s usability in the Evaluation Phase among a cohort of 30 surrogate decision makers using the Systems Usability Scale (SUS). Acceptability was assessed using measures of satisfaction and preference for electronic Collaborative Decision Support (eCODES) versus the original printed decision aid.ResultsThe final decision aid, termed ‘electronic Collaborative Decision Support’, provides a framework for shared decision making, elicits relevant values and preferences, incorporates clinical data to personalize prognostic estimates generated from the ProVent prediction model, generates a printable document summarizing the user’s interaction with the decision aid, and can digitally archive each user session. Usability was excellent (mean SUS, 80 ± 10) overall, but lower among those 56 years and older (73 ± 7) versus those who were younger (84 ± 9); p = 0.03. A total of 93% of users reported a preference for electronic versus printed versions.ConclusionsThe Web-based decision aid for ICU surrogate decision makers can facilitate highly individualized information sharing with excellent usability and acceptability. Decision aids that employ an electronic format such as eCODES represent a strategy that could enhance patient-clinician collaboration and decision making quality in intensive care.Electronic supplementary materialThe online version of this article (doi:10.1186/s13613-015-0045-0) contains supplementary material, which is available to authorized users

Hypercapnia in advanced chronic obstructive pulmonary disease : A secondary analysis of the national emphysema treatment trial

Rationale: Hypercapnia develops in one third of patients with advanced chronic obstructive pulmonary disease (COPD) and is associated with increased morbidity and mortality. Multiple factors in COPD are thought to contribute to the development of hypercapnia including increased carbon dioxide (CO2) production, increased dead space ventilation, and the complex interactions of deranged respiratory system mechanics, inspiratory muscle overload and the ventilatory control center in the brainstem. However, these factors have not previously been systematically analyzed in a large, well-characterized population of severe COPD patients. Methods: This is a secondary analysis of the clinical, physiologic and imaging data from the National Emphysema Treatment Trial (NETT). All patients with complete baseline data for the key predictor variables were included. An inclusive list of 32 potential predictor variables were selected a priori based on consensus of the investigators and literature review. Stepwise variable selection yielded 10 statistically significant associations in multivariate regression. Results: A total of 1419 patients with severe COPD were included in the analysis; mean age 66.4 years (standard deviation 6.3), 38% females, and 422 (29.7%) had baseline hypercapnia. Key variables associated with hypercapnia were low resting partial pressure of oxygen in blood, low minute ventilation (Ve), high volume of exhaled carbon dioxide, low forced expiratory volume in 1 second, high residual volume, lower % emphysema on chest computed tomography, use of oxygen, low ventilatory reserve (high Ve/maximal voluntary ventilation), and not being at high altitude. Low diffusing capacity for carbon monoxide showed a positive association with hypercapnia in univariate analysis but a negative correlation in multivariate analysis. Measures of dyspnea and quality of life did not associate with degree of hypercapnia in multivariable analysis. Conclusions: Hypercapnia in a well-characterized cohort with severe COPD and emphysema is chiefly related to poor lung mechanics, high CO2 production, and a reduced ventilatory capability. Hypercapnia is less impacted by gas exchange abnormalities or the presence of emphysema

A machine learning approach to triaging patients with chronic obstructive pulmonary disease

<div><p>COPD patients are burdened with a daily risk of acute exacerbation and loss of control, which could be mitigated by effective, on-demand decision support tools. In this study, we present a machine learning-based strategy for early detection of exacerbations and subsequent triage. Our application uses physician opinion in a statistically and clinically comprehensive set of patient cases to train a supervised prediction algorithm. The accuracy of the model is assessed against a panel of physicians each triaging identical cases in a representative patient validation set. Our results show that algorithm accuracy and safety indicators surpass all individual pulmonologists in both identifying exacerbations and predicting the consensus triage in a 101 case validation set. The algorithm is also the top performer in sensitivity, specificity, and ppv when predicting a patient’s need for emergency care.</p></div

Distributions for each physician in the validation set (left) and the averaged distributions (right).

<p>(a) triage distribution, (b) averaged triage distribution, (c) exacerbation distribution, (d) averaged exacerbation distribution. Note: error bars indicate 1 standard deviation about the mean.</p

Statistical measures (Eqs 1–7) of triage and exacerbation identification ability for the top 2 performing algorithms and top physician.

<p>Statistical measures (Eqs <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0188532#pone.0188532.e001" target="_blank">1</a>–<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0188532#pone.0188532.e007" target="_blank">7</a>) of triage and exacerbation identification ability for the top 2 performing algorithms and top physician.</p

Performance comparison when the algorithm and all of the physicians got a vote in the consensus opinion.

<p>Comparison of the algorithm and individual physicians at predicting the consensus triage and exacerbation (y/n) in the validation set: (a) triage identification, (b) exacerbation identification. A comparison of the algorithm with the average physician in accuracy, sensitivity, specificity, ppv, and npv for: (c) triage identification, (d) exacerbation identification. Triage statistics were computed as defined in Eqs <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0188532#pone.0188532.e001" target="_blank">1</a>–<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0188532#pone.0188532.e007" target="_blank">7</a>.</p

Feature importance for the top two performing algorithms for both triage and exacerbation models.

<p>Feature importance for the top two performing algorithms for both triage and exacerbation models.</p